{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#########################\n",
    "# Make sure you are using the `conda_python3` Jupyter Kernel for SageMaker!\n",
    "## We will install the necessary libraries.\n",
    "\n",
    "#########################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Training and Serving in SageMaker \"Script Mode\"\n",
    "\n",
    "Script mode is a training script format for TensorFlow that lets you execute any TensorFlow training script in SageMaker with minimal modification. The [SageMaker Python SDK](https://github.com/aws/sagemaker-python-sdk) handles transferring your script to a SageMaker training instance. On the training instance, SageMaker's native TensorFlow support sets up training-related environment variables and executes your training script. In this tutorial, we use the SageMaker Python SDK to launch a training job and deploy the trained model.\n",
    "\n",
    "Script mode supports training with a Python script, a Python module, or a shell script. In this example, we use a Python script to train a classification model on the [MNIST dataset](http://yann.lecun.com/exdb/mnist/). In this example, we will show how easily you can train a SageMaker using TensorFlow 1.x and TensorFlow 2.x scripts with SageMaker Python SDK. In addition, this notebook demonstrates how to perform real time inference with the [SageMaker TensorFlow Serving container](https://github.com/aws/sagemaker-tensorflow-serving-container). The TensorFlow Serving container is the default inference method for script mode. For full documentation on the TensorFlow Serving container, please visit [here](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/deploying_tensorflow_serving.rst).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Set up the environment\n",
    "\n",
    "Let's start by setting up the environment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip uninstall -y tensorflow tensorflow-estimator tb-nightly tf-estimator-nightly tensorboard tensorboardX tensorflow-hub tensorflow-metadata tfds-nightly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip3 uninstall -y tensorflow tensorflow-estimator tb-nightly tf-estimator-nightly tensorboard tensorboardX tensorflow-hub tensorflow-metadata tfds-nightly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip3 install tensorflow==2.0.0b1 --upgrade --ignore-installed --no-cache --user # tensorboard==1.14.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install sagemaker --upgrade --ignore-installed --no-cache --user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install --upgrade --ignore-installed --no-cache stepfunctions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!pip3 install requests==2.20.1 --user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Restart the Kernel to Recognize New Dependencies Above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display_html\n",
    "display_html(\"<script>Jupyter.notebook.kernel.restart()</script>\", raw=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip3 list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add a policy to your SageMaker role in IAM\n",
    "\n",
    "**If you are running this notebook on an Amazon SageMaker notebook instance**, the IAM role assumed by your notebook instance needs permission to create and run workflows in AWS Step Functions. To provide this permission to the role, do the following.\n",
    "\n",
    "1. Open the Amazon [SageMaker console](https://console.aws.amazon.com/sagemaker/). \n",
    "2. Select **Notebook instances** and choose the name of your notebook instance\n",
    "3. Under **Permissions and encryption** select the role ARN to view the role on the IAM console\n",
    "4. Choose **Attach policies** and search for `AWSStepFunctionsFullAccess`.\n",
    "5. Select the check box next to `AWSStepFunctionsFullAccess` and choose **Attach policy**\n",
    "\n",
    "If you are running this notebook in a local environment, the SDK will use your configured AWS CLI configuration. For more information, see [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html).\n",
    "\n",
    "Next, create an execution role in IAM for Step Functions. \n",
    "\n",
    "### Create an execution role for Step Functions\n",
    "\n",
    "You need an execution role so that you can create and execute workflows in Step Functions.\n",
    "\n",
    "1. Go to the [IAM console](https://console.aws.amazon.com/iam/)\n",
    "2. Select **Roles** and then **Create role**.\n",
    "3. Under **Choose the service that will use this role** select **Step Functions**\n",
    "4. Choose **Next** until you can enter a **Role name**\n",
    "5. Enter a name such as `StepFunctionsWorkflowExecutionRole` and then select **Create role**\n",
    "\n",
    "\n",
    "Attach a policy to the role you created. The following steps attach a policy that provides full access to Step Functions, however as a good practice you should only provide access to the resources you need.  \n",
    "\n",
    "1. Under the **Permissions** tab, click **Add inline policy**\n",
    "2. Enter the following in the **JSON** tab\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"sagemaker:CreateTransformJob\",\n",
    "                \"sagemaker:DescribeTransformJob\",\n",
    "                \"sagemaker:StopTransformJob\",\n",
    "                \"sagemaker:CreateTrainingJob\",\n",
    "                \"sagemaker:DescribeTrainingJob\",\n",
    "                \"sagemaker:StopTrainingJob\",\n",
    "                \"sagemaker:CreateHyperParameterTuningJob\",\n",
    "                \"sagemaker:DescribeHyperParameterTuningJob\",\n",
    "                \"sagemaker:StopHyperParameterTuningJob\",\n",
    "                \"sagemaker:CreateModel\",\n",
    "                \"sagemaker:CreateEndpointConfig\",\n",
    "                \"sagemaker:CreateEndpoint\",\n",
    "                \"sagemaker:DeleteEndpointConfig\",\n",
    "                \"sagemaker:DeleteEndpoint\",\n",
    "                \"sagemaker:UpdateEndpoint\",\n",
    "                \"sagemaker:ListTags\",\n",
    "                \"lambda:InvokeFunction\",\n",
    "                \"sqs:SendMessage\",\n",
    "                \"sns:Publish\",\n",
    "                \"ecs:RunTask\",\n",
    "                \"ecs:StopTask\",\n",
    "                \"ecs:DescribeTasks\",\n",
    "                \"dynamodb:GetItem\",\n",
    "                \"dynamodb:PutItem\",\n",
    "                \"dynamodb:UpdateItem\",\n",
    "                \"dynamodb:DeleteItem\",\n",
    "                \"batch:SubmitJob\",\n",
    "                \"batch:DescribeJobs\",\n",
    "                \"batch:TerminateJob\",\n",
    "                \"glue:StartJobRun\",\n",
    "                \"glue:GetJobRun\",\n",
    "                \"glue:GetJobRuns\",\n",
    "                \"glue:BatchStopJobRun\"\n",
    "            ],\n",
    "            \"Resource\": \"*\"\n",
    "        },\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"iam:PassRole\"\n",
    "            ],\n",
    "            \"Resource\": \"*\",\n",
    "            \"Condition\": {\n",
    "                \"StringEquals\": {\n",
    "                    \"iam:PassedToService\": \"sagemaker.amazonaws.com\"\n",
    "                }\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"events:PutTargets\",\n",
    "                \"events:PutRule\",\n",
    "                \"events:DescribeRule\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTrainingJobsRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTransformJobsRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForSageMakerTuningJobsRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForECSTaskRule\",\n",
    "                \"arn:aws:events:*:*:rule/StepFunctionsGetEventsForBatchJobsRule\"\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```\n",
    "\n",
    "3. Choose **Review policy** and give the policy a name such as `StepFunctionsWorkflowExecutionPolicy`\n",
    "4. Choose **Create policy**. You will be redirected to the details page for the role.\n",
    "5. Copy the **Role ARN** at the top of the **Summary**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the required modules & create the AWS SageMaker execution role\n",
    "\n",
    "Now import the required modules from the Step Functions SDK and AWS SageMaker, configure an S3 bucket, and get the AWS SageMaker execution role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sagemaker\n",
    "import stepfunctions\n",
    "import logging\n",
    "\n",
    "from stepfunctions.template.pipeline import TrainingPipeline\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "stepfunctions.set_stream_logger(level=logging.INFO)\n",
    "\n",
    "#bucket = sagemaker_session.default_bucket()\n",
    "#prefix = 'sagemaker/DEMO-tensorflow-mnist'\n",
    "\n",
    "# SageMaker Execution Role\n",
    "# You can use sagemaker.get_execution_role() if running inside sagemaker's notebook instance\n",
    "sagemaker_execution_role = sagemaker.get_execution_role() #Replace with ARN if not in an AWS SageMaker notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup the Service Execution Role and Region\n",
    "Get IAM role arn used to give training and hosting access to your data.  See the documentation for how to create these.  Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the `sagemaker.get_execution_role()` with a the appropriate full IAM role arn string(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RoleARN:  arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881\n",
      "\n",
      "Region:  us-east-1\n"
     ]
    }
   ],
   "source": [
    "#role = get_execution_role()\n",
    "print('RoleARN:  {}\\n'.format(sagemaker_execution_role))\n",
    "\n",
    "region = sagemaker_session.boto_session.region_name\n",
    "print('Region:  {}'.format(region))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Data\n",
    "\n",
    "The MNIST dataset has been loaded to the public S3 buckets ``sagemaker-sample-data-<REGION>`` under the prefix ``tensorflow/mnist``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-sample-data-us-east-1/tensorflow/mnist\n"
     ]
    }
   ],
   "source": [
    "original_training_data_uri = 's3://sagemaker-sample-data-{}/tensorflow/mnist'.format(region)\n",
    "print(original_training_data_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copy the Training Data to Your Notebook Disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "local_data_path = './data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://sagemaker-sample-data-us-east-1/tensorflow/mnist/eval_labels.npy to data/eval_labels.npy\n",
      "download: s3://sagemaker-sample-data-us-east-1/tensorflow/mnist/train_labels.npy to data/train_labels.npy\n",
      "download: s3://sagemaker-sample-data-us-east-1/tensorflow/mnist/eval_data.npy to data/eval_data.npy\n",
      "download: s3://sagemaker-sample-data-us-east-1/tensorflow/mnist/train_data.npy to data/train_data.npy\n"
     ]
    }
   ],
   "source": [
    "!aws --region {region} s3 cp --recursive {original_training_data_uri} {local_data_path}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are four ``.npy`` file under this prefix:\n",
    "* ``train_data.npy``\n",
    "* ``eval_data.npy``\n",
    "* ``train_labels.npy``\n",
    "* ``eval_labels.npy``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "eval_data.npy  eval_labels.npy\ttrain_data.npy\ttrain_labels.npy\r\n"
     ]
    }
   ],
   "source": [
    "!ls {local_data_path}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload the Data to S3 for Distributed Training Across Many Workers\n",
    "We are going to use the `sagemaker.Session.upload_data` function to upload our datasets to an S3 location. The return value inputs identifies the location -- we will use later when we start the training job.\n",
    "\n",
    "This is S3 bucket and prefix that you want to use for training and model data.  This should be within the same region as the Notebook Instance, training, and hosting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket = sagemaker_session.default_bucket()\n",
    "data_prefix = 'sagemaker/tensorflow-mnist/data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/data\n"
     ]
    }
   ],
   "source": [
    "training_data_uri = sagemaker_session.upload_data(path=local_data_path, bucket=bucket, key_prefix=data_prefix)\n",
    "print(training_data_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-02-23 19:14:35   31360128 sagemaker/tensorflow-mnist/data/eval_data.npy\r\n",
      "2020-02-23 19:14:35      40128 sagemaker/tensorflow-mnist/data/eval_labels.npy\r\n",
      "2020-02-23 19:14:35  172480128 sagemaker/tensorflow-mnist/data/train_data.npy\r\n",
      "2020-02-23 19:14:35     220128 sagemaker/tensorflow-mnist/data/train_labels.npy\r\n"
     ]
    }
   ],
   "source": [
    "!aws s3 ls --recursive {training_data_uri}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "https://sagemaker.readthedocs.io/en/stable/using_tf.html#distributed-training\n",
    "\n",
    "This tutorial's training script was adapted from TensorFlow's official [CNN MNIST example](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/layers/cnn_mnist.py). We have modified it to handle the ``model_dir`` parameter passed in by SageMaker. This is an S3 path which can be used for data sharing during distributed training and checkpointing and/or model persistence. We have also added an argument-parsing function to handle processing training-related variables.\n",
    "\n",
    "At the end of the training job we have added a step to export the trained model to the path stored in the environment variable ``SM_MODEL_DIR``, which always points to ``/opt/ml/model``. This is critical because SageMaker uploads all the model artifacts in this folder to S3 at end of training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./src/mnist_keras_tf2.py\r\n"
     ]
    }
   ],
   "source": [
    "!ls ./src/mnist_keras_tf2.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can add custom Python modules to the `src/requirements.txt` file.  They will automatically be installed - and made available to your training script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Python dependencies go here..."
     ]
    }
   ],
   "source": [
    "!cat ./src/requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Step Functions to run training in SageMaker\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train with SageMaker `TensorFlow` Estimator\n",
    "\n",
    "https://sagemaker.readthedocs.io/en/stable/using_tf.html#distributed-training\n",
    "\n",
    "The `sagemaker.tensorflow.TensorFlow` estimator handles locating the script mode container, uploading your script to a S3 location and creating a SageMaker training job. Let's call out a couple important parameters here:\n",
    "\n",
    "* `py_version` is set to `'py3'` to indicate that we are using script mode since legacy mode supports only Python 2. Though Python 2 will be deprecated soon, you can use script mode with Python 2 by setting `py_version` to `'py2'` and `script_mode` to `True`.\n",
    "\n",
    "* `distributions` is used to configure the distributed training setup. It's required only if you are doing distributed training either across a cluster of instances or across multiple GPUs. Here we are using parameter servers as the distributed training schema. SageMaker training jobs run on homogeneous clusters. To make parameter server more performant in the SageMaker setup, we run a parameter server on every instance in the cluster, so there is no need to specify the number of parameter servers to launch. Script mode also supports distributed training with [Horovod](https://github.com/horovod/horovod). You can find the full documentation on how to configure `distributions` [here](https://github.com/aws/sagemaker-python-sdk/tree/master/src/sagemaker/tensorflow#distributed-training). \n",
    "\n",
    "Notes:  \n",
    "* This example uses two(2) `ml.p3.2xlarge` instances.  You will likely need to request a SageMaker instance limit increase from Support before continuing.\n",
    "\n",
    "* Alternatively, you can specify `ml.c5.2xlarge` instead.\n",
    "\n",
    "* To recognize the `requirements.txt`, we must include `src/setup.py` per [this](https://github.com/aws/sagemaker-python-sdk/issues/911) GitHub issue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "# Output Bucket is defined below\n",
    "#model_output_path = 's3://{}/sagemaker/tensorflow-mnist/training-runs'.format(bucket)\n",
    "\n",
    "mnist_estimator = TensorFlow(entry_point='mnist_keras_tf2.py',\n",
    "                             source_dir='./src',\n",
    "#                             output_path=model_output_path,\n",
    "                             role=sagemaker_execution_role,\n",
    "                             train_instance_count=1,\n",
    "                             train_instance_type='ml.c5.2xlarge',\n",
    "                             framework_version='2.0.0',\n",
    "                             py_version='py3',\n",
    "                             enable_sagemaker_metrics=True,\n",
    "                             script_mode=True,\n",
    "                             distributions={'parameter_server': {'enabled': True}},\n",
    "                             # Assuming the loglines from the TensorFlow training job are as follows:\n",
    "                             #    Test loss    : 0.0635053280624561\n",
    "                             #    Test accuracy: 0.9821\n",
    "                             metric_definitions=[\n",
    "                                 {'Name': 'test:loss', 'Regex': 'Test loss    : ([0-9\\\\.]+)'},\n",
    "                                 {'Name': 'test:accuracy', 'Regex': 'Test accuracy: ([0-9\\\\.]+)'},\n",
    "                             ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build a training pipeline with the Step Functions SDK\n",
    "\n",
    "A typical task for a data scientist is to train a model and deploy that model to an endpoint. Without the Step Functions SDK, this is a four step process on SageMaker that includes the following.\n",
    "\n",
    "1. Training the model\n",
    "2. Creating the model on SageMaker\n",
    "3. Creating an endpoint configuration\n",
    "4. Deploying the trained model to the configured endpoint\n",
    "\n",
    "The Step Functions SDK provides the [TrainingPipeline](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/pipelines.html#stepfunctions.template.pipeline.train.TrainingPipeline) API to simplify this procedure. The following configures `pipeline` with the necessary parameters to define a training pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/data\n"
     ]
    }
   ],
   "source": [
    "print(training_data_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# paste the StepFunctionsWorkflowExecutionRole ARN from above\n",
    "workflow_execution_role = \"arn:aws:iam::835319576252:role/StepFunctionsWorkflowExecutionRole\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs\n"
     ]
    }
   ],
   "source": [
    "model_output_s3_bucket = '{}/sagemaker/tensorflow-mnist/training-runs'.format(bucket)\n",
    "print(model_output_s3_bucket)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = TrainingPipeline(\n",
    "    estimator=mnist_estimator,\n",
    "    role=workflow_execution_role,\n",
    "    inputs=training_data_uri,\n",
    "    s3_bucket=model_output_s3_bucket\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize the pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now view the workflow definition, and also visualize it as a graph. This workflow and graph represent your training pipeline.\n",
    "\n",
    "#### View the workflow definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"StartAt\": \"Training\",\n",
      "    \"States\": {\n",
      "        \"Training\": {\n",
      "            \"Resource\": \"arn:aws:states:::sagemaker:createTrainingJob.sync\",\n",
      "            \"Parameters\": {\n",
      "                \"AlgorithmSpecification.$\": \"$$.Execution.Input['Training'].AlgorithmSpecification\",\n",
      "                \"OutputDataConfig.$\": \"$$.Execution.Input['Training'].OutputDataConfig\",\n",
      "                \"StoppingCondition.$\": \"$$.Execution.Input['Training'].StoppingCondition\",\n",
      "                \"ResourceConfig.$\": \"$$.Execution.Input['Training'].ResourceConfig\",\n",
      "                \"RoleArn.$\": \"$$.Execution.Input['Training'].RoleArn\",\n",
      "                \"InputDataConfig.$\": \"$$.Execution.Input['Training'].InputDataConfig\",\n",
      "                \"HyperParameters.$\": \"$$.Execution.Input['Training'].HyperParameters\",\n",
      "                \"TrainingJobName.$\": \"$$.Execution.Input['Training'].TrainingJobName\"\n",
      "            },\n",
      "            \"Type\": \"Task\",\n",
      "            \"Next\": \"Create Model\"\n",
      "        },\n",
      "        \"Create Model\": {\n",
      "            \"Parameters\": {\n",
      "                \"ModelName.$\": \"$$.Execution.Input['Create Model'].ModelName\",\n",
      "                \"PrimaryContainer.$\": \"$$.Execution.Input['Create Model'].PrimaryContainer\",\n",
      "                \"ExecutionRoleArn.$\": \"$$.Execution.Input['Create Model'].ExecutionRoleArn\"\n",
      "            },\n",
      "            \"Resource\": \"arn:aws:states:::sagemaker:createModel\",\n",
      "            \"Type\": \"Task\",\n",
      "            \"Next\": \"Configure Endpoint\"\n",
      "        },\n",
      "        \"Configure Endpoint\": {\n",
      "            \"Resource\": \"arn:aws:states:::sagemaker:createEndpointConfig\",\n",
      "            \"Parameters\": {\n",
      "                \"EndpointConfigName.$\": \"$$.Execution.Input['Configure Endpoint'].EndpointConfigName\",\n",
      "                \"ProductionVariants.$\": \"$$.Execution.Input['Configure Endpoint'].ProductionVariants\"\n",
      "            },\n",
      "            \"Type\": \"Task\",\n",
      "            \"Next\": \"Deploy\"\n",
      "        },\n",
      "        \"Deploy\": {\n",
      "            \"Resource\": \"arn:aws:states:::sagemaker:createEndpoint\",\n",
      "            \"Parameters\": {\n",
      "                \"EndpointConfigName.$\": \"$$.Execution.Input['Deploy'].EndpointConfigName\",\n",
      "                \"EndpointName.$\": \"$$.Execution.Input['Deploy'].EndpointName\"\n",
      "            },\n",
      "            \"Type\": \"Task\",\n",
      "            \"End\": true\n",
      "        }\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "print(pipeline.workflow.definition.to_json(pretty=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visualize the workflow graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://do0of8uwbahzz.cloudfront.net/graph.css\">\n",
       "<div id=\"graph-512\" class=\"workflowgraph\">\n",
       "    \n",
       "    <svg></svg>\n",
       "    \n",
       "</div>\n",
       "\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "require.config({\n",
       "    paths: {\n",
       "        sfn: \"https://do0of8uwbahzz.cloudfront.net/sfn\",\n",
       "    }\n",
       "});\n",
       "\n",
       "require(['sfn'], function(sfn) {\n",
       "    var options = {\n",
       "        width: $('#graph-512').width(),\n",
       "        height: 600,\n",
       "        layout: 'LR',\n",
       "        resizeHeight: true\n",
       "    };\n",
       "\n",
       "    var definition = {\"StartAt\": \"Training\", \"States\": {\"Training\": {\"Resource\": \"arn:aws:states:::sagemaker:createTrainingJob.sync\", \"Parameters\": {\"AlgorithmSpecification.$\": \"$$.Execution.Input['Training'].AlgorithmSpecification\", \"OutputDataConfig.$\": \"$$.Execution.Input['Training'].OutputDataConfig\", \"StoppingCondition.$\": \"$$.Execution.Input['Training'].StoppingCondition\", \"ResourceConfig.$\": \"$$.Execution.Input['Training'].ResourceConfig\", \"RoleArn.$\": \"$$.Execution.Input['Training'].RoleArn\", \"InputDataConfig.$\": \"$$.Execution.Input['Training'].InputDataConfig\", \"HyperParameters.$\": \"$$.Execution.Input['Training'].HyperParameters\", \"TrainingJobName.$\": \"$$.Execution.Input['Training'].TrainingJobName\"}, \"Type\": \"Task\", \"Next\": \"Create Model\"}, \"Create Model\": {\"Parameters\": {\"ModelName.$\": \"$$.Execution.Input['Create Model'].ModelName\", \"PrimaryContainer.$\": \"$$.Execution.Input['Create Model'].PrimaryContainer\", \"ExecutionRoleArn.$\": \"$$.Execution.Input['Create Model'].ExecutionRoleArn\"}, \"Resource\": \"arn:aws:states:::sagemaker:createModel\", \"Type\": \"Task\", \"Next\": \"Configure Endpoint\"}, \"Configure Endpoint\": {\"Resource\": \"arn:aws:states:::sagemaker:createEndpointConfig\", \"Parameters\": {\"EndpointConfigName.$\": \"$$.Execution.Input['Configure Endpoint'].EndpointConfigName\", \"ProductionVariants.$\": \"$$.Execution.Input['Configure Endpoint'].ProductionVariants\"}, \"Type\": \"Task\", \"Next\": \"Deploy\"}, \"Deploy\": {\"Resource\": \"arn:aws:states:::sagemaker:createEndpoint\", \"Parameters\": {\"EndpointConfigName.$\": \"$$.Execution.Input['Deploy'].EndpointConfigName\", \"EndpointName.$\": \"$$.Execution.Input['Deploy'].EndpointName\"}, \"Type\": \"Task\", \"End\": true}}};\n",
       "    var elementId = '#graph-512';\n",
       "\n",
       "    var graph = new sfn.StateMachineGraph(definition, elementId, options);\n",
       "    graph.render();\n",
       "});\n",
       "\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.render_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create and execute the pipeline on AWS Step Functions\n",
    "\n",
    "Create the pipeline in AWS Step Functions with [create](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.html#stepfunctions.workflow.Workflow.create)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[INFO] Workflow created successfully on AWS Step Functions.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'arn:aws:states:us-east-1:835319576252:stateMachine:training-pipeline-2020-02-23-19-14-47'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.create()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the workflow with [execute](https://aws-step-functions-data-science-sdk.readthedocs.io/en/latest/workflow.html#stepfunctions.workflow.Workflow.execute). A link will be provided after the following cell is executed. Following this link, you can monitor your pipeline execution on Step Functions' console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[INFO] Workflow execution started successfully on AWS Step Functions.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "execution = pipeline.execute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://do0of8uwbahzz.cloudfront.net/graph.css\">\n",
       "<div id=\"graph-975\" class=\"workflowgraph\">\n",
       "    \n",
       "    <style>\n",
       "        .graph-legend ul {\n",
       "            list-style-type: none;\n",
       "            padding: 10px;\n",
       "            padding-left: 0;\n",
       "            margin: 0;\n",
       "            position: absolute;\n",
       "            top: 0;\n",
       "            background: transparent;\n",
       "        }\n",
       "\n",
       "        .graph-legend li {\n",
       "            margin-left: 10px;\n",
       "            display: inline-block;\n",
       "        }\n",
       "\n",
       "        .graph-legend li > div {\n",
       "            width: 10px;\n",
       "            height: 10px;\n",
       "            display: inline-block;\n",
       "        }\n",
       "\n",
       "        .graph-legend .success { background-color: #2BD62E }\n",
       "        .graph-legend .failed { background-color: #DE322F }\n",
       "        .graph-legend .cancelled { background-color: #DDDDDD }\n",
       "        .graph-legend .in-progress { background-color: #53C9ED }\n",
       "        .graph-legend .caught-error { background-color: #FFA500 }\n",
       "    </style>\n",
       "    <div class=\"graph-legend\">\n",
       "        <ul>\n",
       "            <li>\n",
       "                <div class=\"success\"></div>\n",
       "                <span>Success</span>\n",
       "            </li>\n",
       "            <li>\n",
       "                <div class=\"failed\"></div>\n",
       "                <span>Failed</span>\n",
       "            </li>\n",
       "            <li>\n",
       "                <div class=\"cancelled\"></div>\n",
       "                <span>Cancelled</span>\n",
       "            </li>\n",
       "            <li>\n",
       "                <div class=\"in-progress\"></div>\n",
       "                <span>In Progress</span>\n",
       "            </li>\n",
       "            <li>\n",
       "                <div class=\"caught-error\"></div>\n",
       "                <span>Caught Error</span>\n",
       "            </li>\n",
       "        </ul>\n",
       "    </div>\n",
       "\n",
       "    <svg></svg>\n",
       "    <a href=\"https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:835319576252:execution:training-pipeline-2020-02-23-19-14-47:training-pipeline-2020-02-23-19-15-03\" target=\"_blank\"> Inspect in AWS Step Functions </a>\n",
       "</div>\n",
       "\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "require.config({\n",
       "    paths: {\n",
       "        sfn: \"https://do0of8uwbahzz.cloudfront.net/sfn\",\n",
       "    }\n",
       "});\n",
       "\n",
       "require(['sfn'], function(sfn) {\n",
       "    var options = {\n",
       "        width: $('#graph-975').width(),\n",
       "        height: 1000,\n",
       "        layout: 'LR',\n",
       "        resizeHeight: true\n",
       "    };\n",
       "\n",
       "    var definition = {\"StartAt\": \"Training\", \"States\": {\"Training\": {\"Resource\": \"arn:aws:states:::sagemaker:createTrainingJob.sync\", \"Parameters\": {\"AlgorithmSpecification.$\": \"$$.Execution.Input['Training'].AlgorithmSpecification\", \"OutputDataConfig.$\": \"$$.Execution.Input['Training'].OutputDataConfig\", \"StoppingCondition.$\": \"$$.Execution.Input['Training'].StoppingCondition\", \"ResourceConfig.$\": \"$$.Execution.Input['Training'].ResourceConfig\", \"RoleArn.$\": \"$$.Execution.Input['Training'].RoleArn\", \"InputDataConfig.$\": \"$$.Execution.Input['Training'].InputDataConfig\", \"HyperParameters.$\": \"$$.Execution.Input['Training'].HyperParameters\", \"TrainingJobName.$\": \"$$.Execution.Input['Training'].TrainingJobName\"}, \"Type\": \"Task\", \"Next\": \"Create Model\"}, \"Create Model\": {\"Parameters\": {\"ModelName.$\": \"$$.Execution.Input['Create Model'].ModelName\", \"PrimaryContainer.$\": \"$$.Execution.Input['Create Model'].PrimaryContainer\", \"ExecutionRoleArn.$\": \"$$.Execution.Input['Create Model'].ExecutionRoleArn\"}, \"Resource\": \"arn:aws:states:::sagemaker:createModel\", \"Type\": \"Task\", \"Next\": \"Configure Endpoint\"}, \"Configure Endpoint\": {\"Resource\": \"arn:aws:states:::sagemaker:createEndpointConfig\", \"Parameters\": {\"EndpointConfigName.$\": \"$$.Execution.Input['Configure Endpoint'].EndpointConfigName\", \"ProductionVariants.$\": \"$$.Execution.Input['Configure Endpoint'].ProductionVariants\"}, \"Type\": \"Task\", \"Next\": \"Deploy\"}, \"Deploy\": {\"Resource\": \"arn:aws:states:::sagemaker:createEndpoint\", \"Parameters\": {\"EndpointConfigName.$\": \"$$.Execution.Input['Deploy'].EndpointConfigName\", \"EndpointName.$\": \"$$.Execution.Input['Deploy'].EndpointName\"}, \"Type\": \"Task\", \"End\": true}}};\n",
       "    var elementId = '#graph-975';\n",
       "    var events = { 'events': [{\"timestamp\": 1582485303.947, \"type\": \"ExecutionStarted\", \"id\": 1, \"previousEventId\": 0, \"executionStartedEventDetails\": {\"input\": \"{\\n    \\\"Training\\\": {\\n        \\\"AlgorithmSpecification\\\": {\\n            \\\"TrainingImage\\\": \\\"763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.0.0-cpu-py3\\\",\\n            \\\"TrainingInputMode\\\": \\\"File\\\"\\n        },\\n        \\\"OutputDataConfig\\\": {\\n            \\\"S3OutputPath\\\": \\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models\\\"\\n        },\\n        \\\"StoppingCondition\\\": {\\n            \\\"MaxRuntimeInSeconds\\\": 86400\\n        },\\n        \\\"ResourceConfig\\\": {\\n            \\\"InstanceCount\\\": 1,\\n            \\\"InstanceType\\\": \\\"ml.c5.2xlarge\\\",\\n            \\\"VolumeSizeInGB\\\": 30\\n        },\\n        \\\"RoleArn\\\": \\\"arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881\\\",\\n        \\\"InputDataConfig\\\": [\\n            {\\n                \\\"DataSource\\\": {\\n                    \\\"S3DataSource\\\": {\\n                        \\\"S3DataType\\\": \\\"S3Prefix\\\",\\n                        \\\"S3Uri\\\": \\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/data\\\",\\n                        \\\"S3DataDistributionType\\\": \\\"FullyReplicated\\\"\\n                    }\\n                },\\n                \\\"ChannelName\\\": \\\"training\\\"\\n            }\\n        ],\\n        \\\"HyperParameters\\\": {\\n            \\\"sagemaker_submit_directory\\\": \\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/source/sourcedir.tar.gz\\\\\\\"\\\",\\n            \\\"sagemaker_program\\\": \\\"\\\\\\\"mnist_keras_tf2.py\\\\\\\"\\\",\\n            \\\"sagemaker_enable_cloudwatch_metrics\\\": \\\"false\\\",\\n            \\\"sagemaker_container_log_level\\\": \\\"20\\\",\\n            \\\"sagemaker_job_name\\\": \\\"\\\\\\\"training-pipeline-2020-02-23-19-14-47/estimator-source\\\\\\\"\\\",\\n            \\\"sagemaker_region\\\": \\\"\\\\\\\"us-east-1\\\\\\\"\\\",\\n            \\\"sagemaker_parameter_server_enabled\\\": \\\"true\\\",\\n            \\\"model_dir\\\": \\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/model\\\\\\\"\\\"\\n        },\\n        \\\"TrainingJobName\\\": \\\"estimator-training-pipeline-2020-02-23-19-15-03\\\"\\n    },\\n    \\\"Create Model\\\": {\\n        \\\"ModelName\\\": \\\"training-pipeline-2020-02-23-19-15-03\\\",\\n        \\\"PrimaryContainer\\\": {\\n            \\\"Image\\\": \\\"763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:2.0-cpu\\\",\\n            \\\"Environment\\\": {\\n                \\\"SAGEMAKER_PROGRAM\\\": null,\\n                \\\"SAGEMAKER_SUBMIT_DIRECTORY\\\": null,\\n                \\\"SAGEMAKER_ENABLE_CLOUDWATCH_METRICS\\\": \\\"false\\\",\\n                \\\"SAGEMAKER_CONTAINER_LOG_LEVEL\\\": \\\"20\\\",\\n                \\\"SAGEMAKER_REGION\\\": \\\"us-east-1\\\"\\n            },\\n            \\\"ModelDataUrl\\\": \\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models/estimator-training-pipeline-2020-02-23-19-15-03/output/model.tar.gz\\\"\\n        },\\n        \\\"ExecutionRoleArn\\\": \\\"arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881\\\"\\n    },\\n    \\\"Configure Endpoint\\\": {\\n        \\\"EndpointConfigName\\\": \\\"training-pipeline-2020-02-23-19-15-03\\\",\\n        \\\"ProductionVariants\\\": [\\n            {\\n                \\\"InitialInstanceCount\\\": 1,\\n                \\\"InstanceType\\\": \\\"ml.c5.2xlarge\\\",\\n                \\\"ModelName\\\": \\\"training-pipeline-2020-02-23-19-15-03\\\",\\n                \\\"VariantName\\\": \\\"AllTraffic\\\"\\n            }\\n        ]\\n    },\\n    \\\"Deploy\\\": {\\n        \\\"EndpointConfigName\\\": \\\"training-pipeline-2020-02-23-19-15-03\\\",\\n        \\\"EndpointName\\\": \\\"training-pipeline-2020-02-23-19-15-03\\\"\\n    }\\n}\", \"roleArn\": \"arn:aws:iam::835319576252:role/StepFunctionsWorkflowExecutionRole\"}}, {\"timestamp\": 1582485303.987, \"type\": \"TaskStateEntered\", \"id\": 2, \"previousEventId\": 0, \"stateEnteredEventDetails\": {\"name\": \"Training\", \"input\": \"{\\n    \\\"Training\\\": {\\n        \\\"AlgorithmSpecification\\\": {\\n            \\\"TrainingImage\\\": \\\"763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.0.0-cpu-py3\\\",\\n            \\\"TrainingInputMode\\\": \\\"File\\\"\\n        },\\n        \\\"OutputDataConfig\\\": {\\n            \\\"S3OutputPath\\\": \\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models\\\"\\n        },\\n        \\\"StoppingCondition\\\": {\\n            \\\"MaxRuntimeInSeconds\\\": 86400\\n        },\\n        \\\"ResourceConfig\\\": {\\n            \\\"InstanceCount\\\": 1,\\n            \\\"InstanceType\\\": \\\"ml.c5.2xlarge\\\",\\n            \\\"VolumeSizeInGB\\\": 30\\n        },\\n        \\\"RoleArn\\\": \\\"arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881\\\",\\n        \\\"InputDataConfig\\\": [\\n            {\\n                \\\"DataSource\\\": {\\n                    \\\"S3DataSource\\\": {\\n                        \\\"S3DataType\\\": \\\"S3Prefix\\\",\\n                        \\\"S3Uri\\\": \\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/data\\\",\\n                        \\\"S3DataDistributionType\\\": \\\"FullyReplicated\\\"\\n                    }\\n                },\\n                \\\"ChannelName\\\": \\\"training\\\"\\n            }\\n        ],\\n        \\\"HyperParameters\\\": {\\n            \\\"sagemaker_submit_directory\\\": \\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/source/sourcedir.tar.gz\\\\\\\"\\\",\\n            \\\"sagemaker_program\\\": \\\"\\\\\\\"mnist_keras_tf2.py\\\\\\\"\\\",\\n            \\\"sagemaker_enable_cloudwatch_metrics\\\": \\\"false\\\",\\n            \\\"sagemaker_container_log_level\\\": \\\"20\\\",\\n            \\\"sagemaker_job_name\\\": \\\"\\\\\\\"training-pipeline-2020-02-23-19-14-47/estimator-source\\\\\\\"\\\",\\n            \\\"sagemaker_region\\\": \\\"\\\\\\\"us-east-1\\\\\\\"\\\",\\n            \\\"sagemaker_parameter_server_enabled\\\": \\\"true\\\",\\n            \\\"model_dir\\\": \\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/model\\\\\\\"\\\"\\n        },\\n        \\\"TrainingJobName\\\": \\\"estimator-training-pipeline-2020-02-23-19-15-03\\\"\\n    },\\n    \\\"Create Model\\\": {\\n        \\\"ModelName\\\": \\\"training-pipeline-2020-02-23-19-15-03\\\",\\n        \\\"PrimaryContainer\\\": {\\n            \\\"Image\\\": \\\"763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:2.0-cpu\\\",\\n            \\\"Environment\\\": {\\n                \\\"SAGEMAKER_PROGRAM\\\": null,\\n                \\\"SAGEMAKER_SUBMIT_DIRECTORY\\\": null,\\n                \\\"SAGEMAKER_ENABLE_CLOUDWATCH_METRICS\\\": \\\"false\\\",\\n                \\\"SAGEMAKER_CONTAINER_LOG_LEVEL\\\": \\\"20\\\",\\n                \\\"SAGEMAKER_REGION\\\": \\\"us-east-1\\\"\\n            },\\n            \\\"ModelDataUrl\\\": \\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models/estimator-training-pipeline-2020-02-23-19-15-03/output/model.tar.gz\\\"\\n        },\\n        \\\"ExecutionRoleArn\\\": \\\"arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881\\\"\\n    },\\n    \\\"Configure Endpoint\\\": {\\n        \\\"EndpointConfigName\\\": \\\"training-pipeline-2020-02-23-19-15-03\\\",\\n        \\\"ProductionVariants\\\": [\\n            {\\n                \\\"InitialInstanceCount\\\": 1,\\n                \\\"InstanceType\\\": \\\"ml.c5.2xlarge\\\",\\n                \\\"ModelName\\\": \\\"training-pipeline-2020-02-23-19-15-03\\\",\\n                \\\"VariantName\\\": \\\"AllTraffic\\\"\\n            }\\n        ]\\n    },\\n    \\\"Deploy\\\": {\\n        \\\"EndpointConfigName\\\": \\\"training-pipeline-2020-02-23-19-15-03\\\",\\n        \\\"EndpointName\\\": \\\"training-pipeline-2020-02-23-19-15-03\\\"\\n    }\\n}\"}}, {\"timestamp\": 1582485303.987, \"type\": \"TaskScheduled\", \"id\": 3, \"previousEventId\": 2, \"taskScheduledEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createTrainingJob.sync\", \"region\": \"us-east-1\", \"parameters\": \"{\\\"HyperParameters\\\":{\\\"sagemaker_submit_directory\\\":\\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/source/sourcedir.tar.gz\\\\\\\"\\\",\\\"sagemaker_program\\\":\\\"\\\\\\\"mnist_keras_tf2.py\\\\\\\"\\\",\\\"sagemaker_enable_cloudwatch_metrics\\\":\\\"false\\\",\\\"sagemaker_container_log_level\\\":\\\"20\\\",\\\"sagemaker_job_name\\\":\\\"\\\\\\\"training-pipeline-2020-02-23-19-14-47/estimator-source\\\\\\\"\\\",\\\"sagemaker_region\\\":\\\"\\\\\\\"us-east-1\\\\\\\"\\\",\\\"sagemaker_parameter_server_enabled\\\":\\\"true\\\",\\\"model_dir\\\":\\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/model\\\\\\\"\\\"},\\\"AlgorithmSpecification\\\":{\\\"TrainingImage\\\":\\\"763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.0.0-cpu-py3\\\",\\\"TrainingInputMode\\\":\\\"File\\\"},\\\"StoppingCondition\\\":{\\\"MaxRuntimeInSeconds\\\":86400},\\\"TrainingJobName\\\":\\\"estimator-training-pipeline-2020-02-23-19-15-03\\\",\\\"OutputDataConfig\\\":{\\\"S3OutputPath\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models\\\"},\\\"ResourceConfig\\\":{\\\"InstanceCount\\\":1,\\\"InstanceType\\\":\\\"ml.c5.2xlarge\\\",\\\"VolumeSizeInGB\\\":30},\\\"InputDataConfig\\\":[{\\\"DataSource\\\":{\\\"S3DataSource\\\":{\\\"S3DataType\\\":\\\"S3Prefix\\\",\\\"S3Uri\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/data\\\",\\\"S3DataDistributionType\\\":\\\"FullyReplicated\\\"}},\\\"ChannelName\\\":\\\"training\\\"}],\\\"RoleArn\\\":\\\"arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881\\\",\\\"Tags\\\":[{\\\"Key\\\":\\\"MANAGED_BY_AWS\\\",\\\"Value\\\":\\\"STARTED_BY_STEP_FUNCTIONS\\\"}]}\"}}, {\"timestamp\": 1582485304.056, \"type\": \"TaskStarted\", \"id\": 4, \"previousEventId\": 3, \"taskStartedEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createTrainingJob.sync\"}}, {\"timestamp\": 1582485304.311, \"type\": \"TaskSubmitted\", \"id\": 5, \"previousEventId\": 4, \"taskSubmittedEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createTrainingJob.sync\", \"output\": \"{\\\"SdkHttpMetadata\\\":{\\\"HttpHeaders\\\":{\\\"Content-Length\\\":\\\"122\\\",\\\"Content-Type\\\":\\\"application/x-amz-json-1.1\\\",\\\"Date\\\":\\\"Sun, 23 Feb 2020 19:15:03 GMT\\\",\\\"x-amzn-RequestId\\\":\\\"6dddff76-403a-4288-8281-99bf1f5dbdaf\\\"},\\\"HttpStatusCode\\\":200},\\\"SdkResponseMetadata\\\":{\\\"RequestId\\\":\\\"6dddff76-403a-4288-8281-99bf1f5dbdaf\\\"},\\\"TrainingJobArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:training-job/estimator-training-pipeline-2020-02-23-19-15-03\\\"}\"}}, {\"timestamp\": 1582485474.992, \"type\": \"TaskSucceeded\", \"id\": 6, \"previousEventId\": 5, \"taskSucceededEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createTrainingJob.sync\", \"output\": \"{\\\"TrainingJobName\\\":\\\"estimator-training-pipeline-2020-02-23-19-15-03\\\",\\\"TrainingJobArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:training-job/estimator-training-pipeline-2020-02-23-19-15-03\\\",\\\"ModelArtifacts\\\":{\\\"S3ModelArtifacts\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models/estimator-training-pipeline-2020-02-23-19-15-03/output/model.tar.gz\\\"},\\\"TrainingJobStatus\\\":\\\"Completed\\\",\\\"SecondaryStatus\\\":\\\"Completed\\\",\\\"HyperParameters\\\":{\\\"sagemaker_container_log_level\\\":\\\"20\\\",\\\"sagemaker_program\\\":\\\"\\\\\\\"mnist_keras_tf2.py\\\\\\\"\\\",\\\"sagemaker_enable_cloudwatch_metrics\\\":\\\"false\\\",\\\"sagemaker_parameter_server_enabled\\\":\\\"true\\\",\\\"sagemaker_region\\\":\\\"\\\\\\\"us-east-1\\\\\\\"\\\",\\\"model_dir\\\":\\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/model\\\\\\\"\\\",\\\"sagemaker_job_name\\\":\\\"\\\\\\\"training-pipeline-2020-02-23-19-14-47/estimator-source\\\\\\\"\\\",\\\"sagemaker_submit_directory\\\":\\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/source/sourcedir.tar.gz\\\\\\\"\\\"},\\\"AlgorithmSpecification\\\":{\\\"TrainingImage\\\":\\\"763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.0.0-cpu-py3\\\",\\\"TrainingInputMode\\\":\\\"FILE\\\"},\\\"RoleArn\\\":\\\"arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881\\\",\\\"InputDataConfig\\\":[{\\\"ChannelName\\\":\\\"training\\\",\\\"DataSource\\\":{\\\"S3DataSource\\\":{\\\"S3DataType\\\":\\\"S3_PREFIX\\\",\\\"S3Uri\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/data\\\",\\\"S3DataDistributionType\\\":\\\"FULLY_REPLICATED\\\"}},\\\"CompressionType\\\":\\\"NONE\\\",\\\"RecordWrapperType\\\":\\\"NONE\\\"}],\\\"OutputDataConfig\\\":{\\\"S3OutputPath\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models\\\"},\\\"ResourceConfig\\\":{\\\"InstanceType\\\":\\\"ml.c5.2xlarge\\\",\\\"InstanceCount\\\":1.0,\\\"VolumeSizeInGB\\\":30.0},\\\"StoppingCondition\\\":{\\\"MaxRuntimeInSeconds\\\":86400.0},\\\"CreationTime\\\":1.582485304284E12,\\\"TrainingStartTime\\\":1.582485410488E12,\\\"TrainingEndTime\\\":1.582485470829E12,\\\"LastModifiedTime\\\":1.582485470829E12,\\\"SecondaryStatusTransitions\\\":[{\\\"Status\\\":\\\"Starting\\\",\\\"StartTime\\\":1.582485304284E12,\\\"EndTime\\\":1.582485410488E12,\\\"StatusMessage\\\":\\\"Preparing the instances for training\\\"},{\\\"Status\\\":\\\"Downloading\\\",\\\"StartTime\\\":1.582485410488E12,\\\"EndTime\\\":1.58248542632E12,\\\"StatusMessage\\\":\\\"Downloading input data\\\"},{\\\"Status\\\":\\\"Training\\\",\\\"StartTime\\\":1.58248542632E12,\\\"EndTime\\\":1.582485464935E12,\\\"StatusMessage\\\":\\\"Training image download completed. Training in progress.\\\"},{\\\"Status\\\":\\\"Uploading\\\",\\\"StartTime\\\":1.582485464935E12,\\\"EndTime\\\":1.582485470829E12,\\\"StatusMessage\\\":\\\"Uploading generated training model\\\"},{\\\"Status\\\":\\\"Completed\\\",\\\"StartTime\\\":1.582485470829E12,\\\"EndTime\\\":1.582485470829E12,\\\"StatusMessage\\\":\\\"Training job completed\\\"}],\\\"Tags\\\":{\\\"AWS_STEP_FUNCTIONS_EXECUTION_ARN\\\":\\\"arn:aws:states:us-east-1:835319576252:execution:training-pipeline-2020-02-23-19-14-47:training-pipeline-2020-02-23-19-15-03\\\",\\\"MANAGED_BY_AWS\\\":\\\"STARTED_BY_STEP_FUNCTIONS\\\"}}\"}}, {\"timestamp\": 1582485474.992, \"type\": \"TaskStateExited\", \"id\": 7, \"previousEventId\": 6, \"stateExitedEventDetails\": {\"name\": \"Training\", \"output\": \"{\\\"TrainingJobName\\\":\\\"estimator-training-pipeline-2020-02-23-19-15-03\\\",\\\"TrainingJobArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:training-job/estimator-training-pipeline-2020-02-23-19-15-03\\\",\\\"ModelArtifacts\\\":{\\\"S3ModelArtifacts\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models/estimator-training-pipeline-2020-02-23-19-15-03/output/model.tar.gz\\\"},\\\"TrainingJobStatus\\\":\\\"Completed\\\",\\\"SecondaryStatus\\\":\\\"Completed\\\",\\\"HyperParameters\\\":{\\\"sagemaker_container_log_level\\\":\\\"20\\\",\\\"sagemaker_program\\\":\\\"\\\\\\\"mnist_keras_tf2.py\\\\\\\"\\\",\\\"sagemaker_enable_cloudwatch_metrics\\\":\\\"false\\\",\\\"sagemaker_parameter_server_enabled\\\":\\\"true\\\",\\\"sagemaker_region\\\":\\\"\\\\\\\"us-east-1\\\\\\\"\\\",\\\"model_dir\\\":\\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/model\\\\\\\"\\\",\\\"sagemaker_job_name\\\":\\\"\\\\\\\"training-pipeline-2020-02-23-19-14-47/estimator-source\\\\\\\"\\\",\\\"sagemaker_submit_directory\\\":\\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/source/sourcedir.tar.gz\\\\\\\"\\\"},\\\"AlgorithmSpecification\\\":{\\\"TrainingImage\\\":\\\"763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.0.0-cpu-py3\\\",\\\"TrainingInputMode\\\":\\\"FILE\\\"},\\\"RoleArn\\\":\\\"arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881\\\",\\\"InputDataConfig\\\":[{\\\"ChannelName\\\":\\\"training\\\",\\\"DataSource\\\":{\\\"S3DataSource\\\":{\\\"S3DataType\\\":\\\"S3_PREFIX\\\",\\\"S3Uri\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/data\\\",\\\"S3DataDistributionType\\\":\\\"FULLY_REPLICATED\\\"}},\\\"CompressionType\\\":\\\"NONE\\\",\\\"RecordWrapperType\\\":\\\"NONE\\\"}],\\\"OutputDataConfig\\\":{\\\"S3OutputPath\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models\\\"},\\\"ResourceConfig\\\":{\\\"InstanceType\\\":\\\"ml.c5.2xlarge\\\",\\\"InstanceCount\\\":1.0,\\\"VolumeSizeInGB\\\":30.0},\\\"StoppingCondition\\\":{\\\"MaxRuntimeInSeconds\\\":86400.0},\\\"CreationTime\\\":1.582485304284E12,\\\"TrainingStartTime\\\":1.582485410488E12,\\\"TrainingEndTime\\\":1.582485470829E12,\\\"LastModifiedTime\\\":1.582485470829E12,\\\"SecondaryStatusTransitions\\\":[{\\\"Status\\\":\\\"Starting\\\",\\\"StartTime\\\":1.582485304284E12,\\\"EndTime\\\":1.582485410488E12,\\\"StatusMessage\\\":\\\"Preparing the instances for training\\\"},{\\\"Status\\\":\\\"Downloading\\\",\\\"StartTime\\\":1.582485410488E12,\\\"EndTime\\\":1.58248542632E12,\\\"StatusMessage\\\":\\\"Downloading input data\\\"},{\\\"Status\\\":\\\"Training\\\",\\\"StartTime\\\":1.58248542632E12,\\\"EndTime\\\":1.582485464935E12,\\\"StatusMessage\\\":\\\"Training image download completed. Training in progress.\\\"},{\\\"Status\\\":\\\"Uploading\\\",\\\"StartTime\\\":1.582485464935E12,\\\"EndTime\\\":1.582485470829E12,\\\"StatusMessage\\\":\\\"Uploading generated training model\\\"},{\\\"Status\\\":\\\"Completed\\\",\\\"StartTime\\\":1.582485470829E12,\\\"EndTime\\\":1.582485470829E12,\\\"StatusMessage\\\":\\\"Training job completed\\\"}],\\\"Tags\\\":{\\\"AWS_STEP_FUNCTIONS_EXECUTION_ARN\\\":\\\"arn:aws:states:us-east-1:835319576252:execution:training-pipeline-2020-02-23-19-14-47:training-pipeline-2020-02-23-19-15-03\\\",\\\"MANAGED_BY_AWS\\\":\\\"STARTED_BY_STEP_FUNCTIONS\\\"}}\"}}, {\"timestamp\": 1582485474.999, \"type\": \"TaskStateEntered\", \"id\": 8, \"previousEventId\": 7, \"stateEnteredEventDetails\": {\"name\": \"Create Model\", \"input\": \"{\\\"TrainingJobName\\\":\\\"estimator-training-pipeline-2020-02-23-19-15-03\\\",\\\"TrainingJobArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:training-job/estimator-training-pipeline-2020-02-23-19-15-03\\\",\\\"ModelArtifacts\\\":{\\\"S3ModelArtifacts\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models/estimator-training-pipeline-2020-02-23-19-15-03/output/model.tar.gz\\\"},\\\"TrainingJobStatus\\\":\\\"Completed\\\",\\\"SecondaryStatus\\\":\\\"Completed\\\",\\\"HyperParameters\\\":{\\\"sagemaker_container_log_level\\\":\\\"20\\\",\\\"sagemaker_program\\\":\\\"\\\\\\\"mnist_keras_tf2.py\\\\\\\"\\\",\\\"sagemaker_enable_cloudwatch_metrics\\\":\\\"false\\\",\\\"sagemaker_parameter_server_enabled\\\":\\\"true\\\",\\\"sagemaker_region\\\":\\\"\\\\\\\"us-east-1\\\\\\\"\\\",\\\"model_dir\\\":\\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/model\\\\\\\"\\\",\\\"sagemaker_job_name\\\":\\\"\\\\\\\"training-pipeline-2020-02-23-19-14-47/estimator-source\\\\\\\"\\\",\\\"sagemaker_submit_directory\\\":\\\"\\\\\\\"s3://sagemaker-us-east-1-835319576252/training-pipeline-2020-02-23-19-14-47/estimator-source/source/sourcedir.tar.gz\\\\\\\"\\\"},\\\"AlgorithmSpecification\\\":{\\\"TrainingImage\\\":\\\"763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-training:2.0.0-cpu-py3\\\",\\\"TrainingInputMode\\\":\\\"FILE\\\"},\\\"RoleArn\\\":\\\"arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881\\\",\\\"InputDataConfig\\\":[{\\\"ChannelName\\\":\\\"training\\\",\\\"DataSource\\\":{\\\"S3DataSource\\\":{\\\"S3DataType\\\":\\\"S3_PREFIX\\\",\\\"S3Uri\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/data\\\",\\\"S3DataDistributionType\\\":\\\"FULLY_REPLICATED\\\"}},\\\"CompressionType\\\":\\\"NONE\\\",\\\"RecordWrapperType\\\":\\\"NONE\\\"}],\\\"OutputDataConfig\\\":{\\\"S3OutputPath\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models\\\"},\\\"ResourceConfig\\\":{\\\"InstanceType\\\":\\\"ml.c5.2xlarge\\\",\\\"InstanceCount\\\":1.0,\\\"VolumeSizeInGB\\\":30.0},\\\"StoppingCondition\\\":{\\\"MaxRuntimeInSeconds\\\":86400.0},\\\"CreationTime\\\":1.582485304284E12,\\\"TrainingStartTime\\\":1.582485410488E12,\\\"TrainingEndTime\\\":1.582485470829E12,\\\"LastModifiedTime\\\":1.582485470829E12,\\\"SecondaryStatusTransitions\\\":[{\\\"Status\\\":\\\"Starting\\\",\\\"StartTime\\\":1.582485304284E12,\\\"EndTime\\\":1.582485410488E12,\\\"StatusMessage\\\":\\\"Preparing the instances for training\\\"},{\\\"Status\\\":\\\"Downloading\\\",\\\"StartTime\\\":1.582485410488E12,\\\"EndTime\\\":1.58248542632E12,\\\"StatusMessage\\\":\\\"Downloading input data\\\"},{\\\"Status\\\":\\\"Training\\\",\\\"StartTime\\\":1.58248542632E12,\\\"EndTime\\\":1.582485464935E12,\\\"StatusMessage\\\":\\\"Training image download completed. Training in progress.\\\"},{\\\"Status\\\":\\\"Uploading\\\",\\\"StartTime\\\":1.582485464935E12,\\\"EndTime\\\":1.582485470829E12,\\\"StatusMessage\\\":\\\"Uploading generated training model\\\"},{\\\"Status\\\":\\\"Completed\\\",\\\"StartTime\\\":1.582485470829E12,\\\"EndTime\\\":1.582485470829E12,\\\"StatusMessage\\\":\\\"Training job completed\\\"}],\\\"Tags\\\":{\\\"AWS_STEP_FUNCTIONS_EXECUTION_ARN\\\":\\\"arn:aws:states:us-east-1:835319576252:execution:training-pipeline-2020-02-23-19-14-47:training-pipeline-2020-02-23-19-15-03\\\",\\\"MANAGED_BY_AWS\\\":\\\"STARTED_BY_STEP_FUNCTIONS\\\"}}\"}}, {\"timestamp\": 1582485474.999, \"type\": \"TaskScheduled\", \"id\": 9, \"previousEventId\": 8, \"taskScheduledEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createModel\", \"region\": \"us-east-1\", \"parameters\": \"{\\\"ExecutionRoleArn\\\":\\\"arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881\\\",\\\"PrimaryContainer\\\":{\\\"Image\\\":\\\"763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:2.0-cpu\\\",\\\"Environment\\\":{\\\"SAGEMAKER_PROGRAM\\\":null,\\\"SAGEMAKER_SUBMIT_DIRECTORY\\\":null,\\\"SAGEMAKER_ENABLE_CLOUDWATCH_METRICS\\\":\\\"false\\\",\\\"SAGEMAKER_CONTAINER_LOG_LEVEL\\\":\\\"20\\\",\\\"SAGEMAKER_REGION\\\":\\\"us-east-1\\\"},\\\"ModelDataUrl\\\":\\\"s3://sagemaker-us-east-1-835319576252/sagemaker/tensorflow-mnist/training-runs/training-pipeline-2020-02-23-19-14-47/models/estimator-training-pipeline-2020-02-23-19-15-03/output/model.tar.gz\\\"},\\\"ModelName\\\":\\\"training-pipeline-2020-02-23-19-15-03\\\"}\"}}, {\"timestamp\": 1582485475.013, \"type\": \"TaskStarted\", \"id\": 10, \"previousEventId\": 9, \"taskStartedEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createModel\"}}, {\"timestamp\": 1582485475.284, \"type\": \"TaskSucceeded\", \"id\": 11, \"previousEventId\": 10, \"taskSucceededEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createModel\", \"output\": \"{\\\"ModelArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:model/training-pipeline-2020-02-23-19-15-03\\\",\\\"SdkHttpMetadata\\\":{\\\"HttpHeaders\\\":{\\\"Content-Length\\\":\\\"99\\\",\\\"Content-Type\\\":\\\"application/x-amz-json-1.1\\\",\\\"Date\\\":\\\"Sun, 23 Feb 2020 19:17:54 GMT\\\",\\\"x-amzn-RequestId\\\":\\\"359496b7-508a-46a4-80fa-06efb3cc6404\\\"},\\\"HttpStatusCode\\\":200},\\\"SdkResponseMetadata\\\":{\\\"RequestId\\\":\\\"359496b7-508a-46a4-80fa-06efb3cc6404\\\"}}\"}}, {\"timestamp\": 1582485475.284, \"type\": \"TaskStateExited\", \"id\": 12, \"previousEventId\": 11, \"stateExitedEventDetails\": {\"name\": \"Create Model\", \"output\": \"{\\\"ModelArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:model/training-pipeline-2020-02-23-19-15-03\\\",\\\"SdkHttpMetadata\\\":{\\\"HttpHeaders\\\":{\\\"Content-Length\\\":\\\"99\\\",\\\"Content-Type\\\":\\\"application/x-amz-json-1.1\\\",\\\"Date\\\":\\\"Sun, 23 Feb 2020 19:17:54 GMT\\\",\\\"x-amzn-RequestId\\\":\\\"359496b7-508a-46a4-80fa-06efb3cc6404\\\"},\\\"HttpStatusCode\\\":200},\\\"SdkResponseMetadata\\\":{\\\"RequestId\\\":\\\"359496b7-508a-46a4-80fa-06efb3cc6404\\\"}}\"}}, {\"timestamp\": 1582485475.291, \"type\": \"TaskStateEntered\", \"id\": 13, \"previousEventId\": 12, \"stateEnteredEventDetails\": {\"name\": \"Configure Endpoint\", \"input\": \"{\\\"ModelArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:model/training-pipeline-2020-02-23-19-15-03\\\",\\\"SdkHttpMetadata\\\":{\\\"HttpHeaders\\\":{\\\"Content-Length\\\":\\\"99\\\",\\\"Content-Type\\\":\\\"application/x-amz-json-1.1\\\",\\\"Date\\\":\\\"Sun, 23 Feb 2020 19:17:54 GMT\\\",\\\"x-amzn-RequestId\\\":\\\"359496b7-508a-46a4-80fa-06efb3cc6404\\\"},\\\"HttpStatusCode\\\":200},\\\"SdkResponseMetadata\\\":{\\\"RequestId\\\":\\\"359496b7-508a-46a4-80fa-06efb3cc6404\\\"}}\"}}, {\"timestamp\": 1582485475.291, \"type\": \"TaskScheduled\", \"id\": 14, \"previousEventId\": 13, \"taskScheduledEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createEndpointConfig\", \"region\": \"us-east-1\", \"parameters\": \"{\\\"ProductionVariants\\\":[{\\\"InitialInstanceCount\\\":1,\\\"InstanceType\\\":\\\"ml.c5.2xlarge\\\",\\\"ModelName\\\":\\\"training-pipeline-2020-02-23-19-15-03\\\",\\\"VariantName\\\":\\\"AllTraffic\\\"}],\\\"EndpointConfigName\\\":\\\"training-pipeline-2020-02-23-19-15-03\\\"}\"}}, {\"timestamp\": 1582485475.302, \"type\": \"TaskStarted\", \"id\": 15, \"previousEventId\": 14, \"taskStartedEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createEndpointConfig\"}}, {\"timestamp\": 1582485475.373, \"type\": \"TaskSucceeded\", \"id\": 16, \"previousEventId\": 15, \"taskSucceededEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createEndpointConfig\", \"output\": \"{\\\"EndpointConfigArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:endpoint-config/training-pipeline-2020-02-23-19-15-03\\\",\\\"SdkHttpMetadata\\\":{\\\"HttpHeaders\\\":{\\\"Content-Length\\\":\\\"118\\\",\\\"Content-Type\\\":\\\"application/x-amz-json-1.1\\\",\\\"Date\\\":\\\"Sun, 23 Feb 2020 19:17:54 GMT\\\",\\\"x-amzn-RequestId\\\":\\\"a391e92f-dcb8-4e72-907b-e39fe4b5a5b8\\\"},\\\"HttpStatusCode\\\":200},\\\"SdkResponseMetadata\\\":{\\\"RequestId\\\":\\\"a391e92f-dcb8-4e72-907b-e39fe4b5a5b8\\\"}}\"}}, {\"timestamp\": 1582485475.373, \"type\": \"TaskStateExited\", \"id\": 17, \"previousEventId\": 16, \"stateExitedEventDetails\": {\"name\": \"Configure Endpoint\", \"output\": \"{\\\"EndpointConfigArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:endpoint-config/training-pipeline-2020-02-23-19-15-03\\\",\\\"SdkHttpMetadata\\\":{\\\"HttpHeaders\\\":{\\\"Content-Length\\\":\\\"118\\\",\\\"Content-Type\\\":\\\"application/x-amz-json-1.1\\\",\\\"Date\\\":\\\"Sun, 23 Feb 2020 19:17:54 GMT\\\",\\\"x-amzn-RequestId\\\":\\\"a391e92f-dcb8-4e72-907b-e39fe4b5a5b8\\\"},\\\"HttpStatusCode\\\":200},\\\"SdkResponseMetadata\\\":{\\\"RequestId\\\":\\\"a391e92f-dcb8-4e72-907b-e39fe4b5a5b8\\\"}}\"}}, {\"timestamp\": 1582485475.401, \"type\": \"TaskStateEntered\", \"id\": 18, \"previousEventId\": 17, \"stateEnteredEventDetails\": {\"name\": \"Deploy\", \"input\": \"{\\\"EndpointConfigArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:endpoint-config/training-pipeline-2020-02-23-19-15-03\\\",\\\"SdkHttpMetadata\\\":{\\\"HttpHeaders\\\":{\\\"Content-Length\\\":\\\"118\\\",\\\"Content-Type\\\":\\\"application/x-amz-json-1.1\\\",\\\"Date\\\":\\\"Sun, 23 Feb 2020 19:17:54 GMT\\\",\\\"x-amzn-RequestId\\\":\\\"a391e92f-dcb8-4e72-907b-e39fe4b5a5b8\\\"},\\\"HttpStatusCode\\\":200},\\\"SdkResponseMetadata\\\":{\\\"RequestId\\\":\\\"a391e92f-dcb8-4e72-907b-e39fe4b5a5b8\\\"}}\"}}, {\"timestamp\": 1582485475.401, \"type\": \"TaskScheduled\", \"id\": 19, \"previousEventId\": 18, \"taskScheduledEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createEndpoint\", \"region\": \"us-east-1\", \"parameters\": \"{\\\"EndpointName\\\":\\\"training-pipeline-2020-02-23-19-15-03\\\",\\\"EndpointConfigName\\\":\\\"training-pipeline-2020-02-23-19-15-03\\\"}\"}}, {\"timestamp\": 1582485475.414, \"type\": \"TaskStarted\", \"id\": 20, \"previousEventId\": 19, \"taskStartedEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createEndpoint\"}}, {\"timestamp\": 1582485475.594, \"type\": \"TaskSucceeded\", \"id\": 21, \"previousEventId\": 20, \"taskSucceededEventDetails\": {\"resourceType\": \"sagemaker\", \"resource\": \"createEndpoint\", \"output\": \"{\\\"EndpointArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:endpoint/training-pipeline-2020-02-23-19-15-03\\\",\\\"SdkHttpMetadata\\\":{\\\"HttpHeaders\\\":{\\\"Content-Length\\\":\\\"105\\\",\\\"Content-Type\\\":\\\"application/x-amz-json-1.1\\\",\\\"Date\\\":\\\"Sun, 23 Feb 2020 19:17:54 GMT\\\",\\\"x-amzn-RequestId\\\":\\\"8ed6f43f-5389-47a7-b5aa-1efe325f1b45\\\"},\\\"HttpStatusCode\\\":200},\\\"SdkResponseMetadata\\\":{\\\"RequestId\\\":\\\"8ed6f43f-5389-47a7-b5aa-1efe325f1b45\\\"}}\"}}, {\"timestamp\": 1582485475.594, \"type\": \"TaskStateExited\", \"id\": 22, \"previousEventId\": 21, \"stateExitedEventDetails\": {\"name\": \"Deploy\", \"output\": \"{\\\"EndpointArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:endpoint/training-pipeline-2020-02-23-19-15-03\\\",\\\"SdkHttpMetadata\\\":{\\\"HttpHeaders\\\":{\\\"Content-Length\\\":\\\"105\\\",\\\"Content-Type\\\":\\\"application/x-amz-json-1.1\\\",\\\"Date\\\":\\\"Sun, 23 Feb 2020 19:17:54 GMT\\\",\\\"x-amzn-RequestId\\\":\\\"8ed6f43f-5389-47a7-b5aa-1efe325f1b45\\\"},\\\"HttpStatusCode\\\":200},\\\"SdkResponseMetadata\\\":{\\\"RequestId\\\":\\\"8ed6f43f-5389-47a7-b5aa-1efe325f1b45\\\"}}\"}}, {\"timestamp\": 1582485475.594, \"type\": \"ExecutionSucceeded\", \"id\": 23, \"previousEventId\": 22, \"executionSucceededEventDetails\": {\"output\": \"{\\\"EndpointArn\\\":\\\"arn:aws:sagemaker:us-east-1:835319576252:endpoint/training-pipeline-2020-02-23-19-15-03\\\",\\\"SdkHttpMetadata\\\":{\\\"HttpHeaders\\\":{\\\"Content-Length\\\":\\\"105\\\",\\\"Content-Type\\\":\\\"application/x-amz-json-1.1\\\",\\\"Date\\\":\\\"Sun, 23 Feb 2020 19:17:54 GMT\\\",\\\"x-amzn-RequestId\\\":\\\"8ed6f43f-5389-47a7-b5aa-1efe325f1b45\\\"},\\\"HttpStatusCode\\\":200},\\\"SdkResponseMetadata\\\":{\\\"RequestId\\\":\\\"8ed6f43f-5389-47a7-b5aa-1efe325f1b45\\\"}}\"}}] };\n",
       "\n",
       "    var graph = new sfn.StateMachineExecutionGraph(definition, events, elementId, options);\n",
       "    graph.render();\n",
       "});\n",
       "\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "execution.render_progress()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "events = execution.list_events()\n",
    "\n",
    "endpoint_name = json.loads(events[18]['taskScheduledEventDetails']['parameters'])['EndpointName']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'EndpointArn': 'arn:aws:sagemaker:us-east-1:835319576252:endpoint/training-pipeline-2020-02-23-19-15-03',\n",
       " 'SdkHttpMetadata': {'HttpHeaders': {'Content-Length': '105',\n",
       "   'Content-Type': 'application/x-amz-json-1.1',\n",
       "   'Date': 'Sun, 23 Feb 2020 19:17:54 GMT',\n",
       "   'x-amzn-RequestId': '8ed6f43f-5389-47a7-b5aa-1efe325f1b45'},\n",
       "  'HttpStatusCode': 200},\n",
       " 'SdkResponseMetadata': {'RequestId': '8ed6f43f-5389-47a7-b5aa-1efe325f1b45'}}"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "event_output = json.loads(events[21]['stateExitedEventDetails']['output'])\n",
    "event_output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_arn = event_output['EndpointArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO:  Retieve the predictor from the pipeline/workflow above\n",
    "# predictor = mnist_estimator.deploy(initial_instance_count=1, instance_type='ml.c5.2xlarge')\n",
    "\n",
    "predictor = sagemaker.predictor.RealTimePredictor(endpoint=endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Invoke the Endpoint\n",
    "\n",
    "Let's download the training data and use that as input for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "train_data = np.load('{}/train_data.npy'.format(local_data_path))\n",
    "train_labels = np.load('{}/train_labels.npy'.format(local_data_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The formats of the input and the output data correspond directly to the request and response formats of the `Predict` method in the [TensorFlow Serving REST API](https://www.tensorflow.org/serving/api_rest). SageMaker's TensforFlow Serving endpoints can also accept additional input formats that are not part of the TensorFlow REST API, including the simplified JSON format, line-delimited JSON objects (\"jsons\" or \"jsonlines\"), and CSV data.\n",
    "\n",
    "In this example we are using a `numpy` array as input, which will be serialized into the simplified JSON format. In addition, TensorFlow Serving can also process multiple items at once as you can see in the following code. You can find the complete documentation on how to make predictions against a TensorFlow Serving SageMaker endpoint [here](https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/tensorflow/deploying_tensorflow_serving.rst#making-predictions-against-a-sagemaker-endpoint)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine the prediction result from the TensorFlow 2.0 model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "ename": "ParamValidationError",
     "evalue": "Parameter validation failed:\nInvalid type for parameter Body, value: [[0. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]\n ...\n [0. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]], type: <class 'numpy.ndarray'>, valid types: <class 'bytes'>, <class 'bytearray'>, file-like object",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mParamValidationError\u001b[0m                      Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-100-55b3c5b086a0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mpredictions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpredictor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_data\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m50\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m50\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m     \u001b[0mprediction\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpredictions\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'predictions'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0mlabel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_labels\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Prediction is {}, Label is {}, Matched: {}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprediction\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprediction\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mlabel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.6/site-packages/sagemaker/predictor.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, data, initial_args, target_model)\u001b[0m\n\u001b[1;32m    108\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    109\u001b[0m         \u001b[0mrequest_args\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_request_args\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minitial_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_model\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 110\u001b[0;31m         \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_runtime_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minvoke_endpoint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mrequest_args\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    111\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_handle_response\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    112\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.6/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_api_call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    314\u001b[0m                     \"%s() only accepts keyword arguments.\" % py_operation_name)\n\u001b[1;32m    315\u001b[0m             \u001b[0;31m# The \"self\" in this scope is referring to the BaseClient.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 316\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_api_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    317\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    318\u001b[0m         \u001b[0m_api_call\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpy_operation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.6/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_make_api_call\u001b[0;34m(self, operation_name, api_params)\u001b[0m\n\u001b[1;32m    597\u001b[0m         }\n\u001b[1;32m    598\u001b[0m         request_dict = self._convert_to_request_dict(\n\u001b[0;32m--> 599\u001b[0;31m             api_params, operation_model, context=request_context)\n\u001b[0m\u001b[1;32m    600\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    601\u001b[0m         \u001b[0mservice_id\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_service_model\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mservice_id\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhyphenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.6/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_convert_to_request_dict\u001b[0;34m(self, api_params, operation_model, context)\u001b[0m\n\u001b[1;32m    645\u001b[0m             api_params, operation_model, context)\n\u001b[1;32m    646\u001b[0m         request_dict = self._serializer.serialize_to_request(\n\u001b[0;32m--> 647\u001b[0;31m             api_params, operation_model)\n\u001b[0m\u001b[1;32m    648\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_client_config\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minject_host_prefix\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    649\u001b[0m             \u001b[0mrequest_dict\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'host_prefix'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.6/site-packages/botocore/validate.py\u001b[0m in \u001b[0;36mserialize_to_request\u001b[0;34m(self, parameters, operation_model)\u001b[0m\n\u001b[1;32m    295\u001b[0m                                                     operation_model.input_shape)\n\u001b[1;32m    296\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mreport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhas_errors\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 297\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mParamValidationError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreport\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgenerate_report\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    298\u001b[0m         return self._serializer.serialize_to_request(parameters,\n\u001b[1;32m    299\u001b[0m                                                      operation_model)\n",
      "\u001b[0;31mParamValidationError\u001b[0m: Parameter validation failed:\nInvalid type for parameter Body, value: [[0. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]\n ...\n [0. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]], type: <class 'numpy.ndarray'>, valid types: <class 'bytes'>, <class 'bytearray'>, file-like object"
     ]
    }
   ],
   "source": [
    "predictions = predictor.predict(train_data[:50])\n",
    "for i in range(0, 50):\n",
    "    prediction = predictions['predictions'][i]\n",
    "    label = train_labels[i]\n",
    "    print('Prediction is {}, Label is {}, Matched: {}'.format(prediction, label, prediction == label))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
